texts:
I have a data set of house prices and their corresponding features (rooms, meter squared, etc). An additional feature is the sold date of the house. The aim is to create a model that can estimate the price of a house as if it was sold today. For example a house with a specific set of features (5 rooms, 100 meters squared) and today's date (28-1-2020), what would it sell for? Time is an important component, because prices increase (inflate over time). I am struggling to find a way to incorporate the sold date as a feature in the gradient boosting model.
I think there are a number of approaches:

Convert the data into an integer, and include it directly in the model as a feature.
Create a separate model for modelling the house price development over time. Let's think of this as some kind of an AR(1) model. I could then adjust all observations for inflation, so that we would get an inflation adjusted price for today. These inflation adjusted prices would be trained on the feature set.

What are your thoughts on these two options? Are there any alternative methods?


texts: Feature importance and deriving rules using tree based classification models

texts:
I have a dataset where I have categorical and continuous values with targets 0/1 (binary classification task). Since I need to find patterns and relationships in the occurrence of the event or target, I think I should use Decision trees. However the issue is that I have 2 categorical variables which have 700 & 150 categories respectively with the remaining variables being numerical/continuous.

My questions are as follows:

Can I estimate feature importance using random forests in sklearn for this set of variables? If yes, do I need to change the type of variables?
Can I use the variables as they are, i.e categorical with so many categories each & continuous variables, when training a decision tree using sklearn in order to be able to visualize rules from the tree?

Please feel free to suggest any other approach too. The prime objective of the task is to identify the drivers/features which impact the target & generate a pattern - if at all can be identified from the data.

Also, given these many categories for the 2 variables, how do I conduct a statistical analysis for this variables alone with the target to understand if any correlation exists or identify cause effect relationships.


texts: Attribute error when loading pickle file in python jupyter notebook

texts:
I have created a config file with the following code for an object detection task and saved in the local disk.

# Create the config
C = Config()
C.use_horizontal_flips = horizontal_flips
C.use_vertical_flips = vertical_flips
C.rot_90 = rot_90
C.record_path = record_path
C.model_path = output_weight_path
C.num_rois = num_rois
C.base_net_weights = base_weight_path

with open(config_output_filename, 'wb') as config_f:
pickle.dump(C,config_f)

I am trying to load this pickle file in another jupyter notebook.

with open(config_output_filename, "rb") as f:
C = pickle.load(f)
# turn off any data augmentation at test time
C.use_horizontal_flips = False
C.use_vertical_flips = False
C.rot_90 = False
print(C.num_rois)

But it gives me the following error.

------------------------------------------------------------------------

AttributeError             Traceback (most recent call
last)
 <ipython-input-20-c5528d5ef98b> in <module>
  1 with open(config_output_filename, "rb") as f:
   ----> 2     C = pickle.load(f)

  AttributeError: Can't get attribute 'Config' on <module '__main__'>

But I was able to load the pickle file from the disk without errors in my previous attempts and applied the configurations on the test set.


texts:
I use a linear regression to predict house prices (https://www.kaggle.com/c/house-prices-advanced-regression-techniques/overview). My linear regression sometimes works great with R^2 of 0.8 and sometimes really sucks with R^2 of - 20000000000000 (Yes, really that bad). My data is scaled (with Min-Max) but my target value isn't. The problem appeared before scaling as well. Apparently my predictions are only a few times off, but then they are really off.
Here is my code:
df = pd.read_csv('data/results/house_prices_advanced_regression/house_prices_advanced_regression_.csv')
target = df['SalePrice']
df = df.drop('SalePrice', axis=1)
x_train, x_test, y_train, y_test = train_test_split(df, target, test_size=0.2)

# model
lr = LinearRegression().fit(x_train, y_train)

predictions = lr.predict(x_test)
rmse = mean_squared_error(y_test, predictions, squared=False)
mae = mean_absolute_error(y_test, predictions)
print('RMSE: {}'.format(rmse))
print('MAE: {}'.format(mae))
x = range(len(y_test))
plt.plot(x, predictions, 'r')
plt.plot(x, y_test, 'b')
print("R^2: {}".format(lr.score(x_test, y_test)))

Output:
'RMSE: 4.2834860573491624e+16'
'MAE: 3483821398471256.5'
'R^2: -3.358286170039772e+23'

Difference between prediction (red) and real value(blue):

Coefficients:

Does anybody know what to do?


texts: How does imblearn apply the transformations during prediction?

texts:
Let's say I have a sklearn pipeline that:

Imputes the data
Randomly oversamples the minority class

from imblearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from imblearn.over_sampling import RandomOverSampler

pipeline = Pipeline(
    [('1', SimpleImputer(strategy='median'),
     ('2', RandomOverSampler(random_state=0)),
     ('estimator', <Some Logistic Regression>)
    ]
)

I can then fit this to my training set pipeline.fit(X_train, y_train) and the random oversampler should properly identify the class to sample. What if I try to predict i.e pipeline.predict(X_test)? Since there are no classes, does the random oversampler still apply? I would expect the imputer to apply regardless but what about RandomOverSampler?

Thank you


texts: Using the input record's distribution as a composite input feature

texts:
I have a very interesting dataset that I need to use for doing regression. It is production data from stainless steel production and I have about 290 input features, so I need to start reducing the dimensions.

Most of the data is batch-type, ie. there is a single record for every batch. However, some of the sub-processes have multiple cycles (that are repeated n times). On each cycle the data is measured as time-series, but the final measurement is only reported with its statistics for cycle n: mean, standard deviation, max and min. The difference here is that you have a distribution for every single record, not for the entire input variable only.

Is there a recommended approach of using distributions on your input records to your advantage?

If I could use these input records as a distribution as inputs to my selected model, I could essentially reduce the number of variables by a factor of 4 (for these sub-processes only). I looked here, here, here, but the closest question and answer was here. It does not answer my question though.

P.S. I have already trained an ANN using the 290 input features (and a suitable dimensionality reduction), but I kept these variables as separate inputs and got good results. I just think there should be a better way, since these statistics for each parameter are for the same parameter, ie. not independent.

If the above is not clear, here is the hierarchy of the data:

For each batch
There is a sub-process
That goes through multiple cycles where the physical parameters are measured in real-time
Each cycle has statistics for the physical parameters (mean, standard deviation, max and min)
Eg. Sub-process X -> Cycle n -> parameter1_mean, parameter1_stddev, parameter1_max, parameter1_min; parameter2_mean, parameter2_stddev, etc


texts:
I have two files :
Test_data  - contains the features of a dataset to find predictions for
Submission_data - contains two columns : The index column for test data and another column for its corresponding predicted value
So , I have to make predictions on the test data and store the predicted values in the submission file.
During preprocessing of the test data , I am dropping rows that do not contain values (NaN) for atleast 50% of the features(columns) :
test_data = test_data.dropna(thresh=math.ceil(test_data.shape[1]/2))

Now , How do I remove the corresponding rows in the submissions dataframe ? Because , If I drop some rows in the test data , I cannot make a prediction for the corresponding row in the submissions dataframe/file.
The problem is , there is an Index column that does NOT HAVE UNIQUE values (In both test data and submissions data)
So , How do I drop the rows in Submissions data that were also dropped in Test data ?
I am new to ML challenges and I find this challenging .


texts: Why is there such a wide range of prices for whole genome sequencing services?

texts:
Below is a list of labs around the country that offer genetic sequencing.  What would be the reasons why they each cost a different amount if the end result is that you get a 90GB hard drive with 3+ billion base pairs for your genome?

https://www.scienceexchange.com/services/whole-genome-seq


texts: How to calculate mAP for detection task for the PASCAL VOC Challenge?

texts:
How to calculate the mAP (mean Average Precision) for the detection task for the Pascal VOC leaderboards?
There said - at page 11:

Average Precision (AP). For the VOC2007 challenge, the interpolated
average precision (Salton and Mcgill 1986) was used to evaluate both
classification and detection. For a given task and class, the
precision/recall curve is computed from a method’s ranked output.
Recall is defined as the proportion of all positive examples ranked
above a given rank. Precision is the proportion of all examples above
that rank which are from the positive class. The AP summarises the
shape of the precision/recall curve, and is defined as the mean
precision at a set of eleven equally spaced recall levels
[0,0.1,...,1]:
AP = 1/11 ∑ r∈{0,0.1,...,1} pinterp(r)
The precision at each recall level r is interpolated by taking the
maximum precision measured for a method for which the corresponding
recall exceeds r: pinterp(r) = max p(r˜), where p(r˜) is the measured
precision at recall ˜r

About mAP

So does it mean that:

We calculate Precision and Recall:

A) For many different IoU > {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1} we calculate True/False Positive/Negative values
Where True positive = Number_of_detection with IoU > {0, 0.1,..., 1}, as said here and then we calculate:
Precision = True positive / (True positive + False positive)
Recall = True positive / (True positive + False negative)

B) Or for many different thresholds of detection algorithms we calculate:
Precision = True positive / (True positive + False positive)
Recall = True positive / (True positive + False negative)
Where True positive = Number_of_detection with IoU > 0.5 as said here

C) Or for many different thresholds of detection algorithms we calculate:
Precision = Intersect / Detected_box
Recall = Intersect / Object
As shown here?

Then we build Precision-Recall curve, as shown here:

Then we calculate AP (average precision) as average of 11 values of Precision at the points where Recall = {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1}, i.e. AP = 1/11 ∑ recall∈{0,0.1,...,1} Precision(Recall)

(In general for each point, for example 0.3, we get MAX of Precision for Recall <= 0.3, instead of value of Precision at this point Recall=0.3)

And when we calculate AP only for 1 something object class on all images - then we get AP (average precision) for this class, for example, only for air.

So AP is a  integral (area under the curve)
But when we calculate AP for all object classes on all images - then we get mAP (mean average precision) for all images dataset.

Questions:

Is it right, and if it isn't, then how to calculate mAP for Pascal VOC Challenge?
And which of the 3 formulas (A, B or C) is correct for calculating Precision and Recall, in paragraph 1?

Short answer:

mAP = AVG(AP for each object class)
AP = AVG(Precision for each of 11 Recalls {precision = 0, 0.1, ..., 1})
PR-curve = Precision and Recall (for each Threshold that is in the Predictions bound-boxes)
Precision = TP / (TP + FP)
Recall = TP / (TP + FN)
TP = number of detections with IoU>0.5
FP = number of detections with IoU<=0.5 or detected more than once
FN = number of objects that not detected or detected with IoU<=0.5


texts:
Hi I would like to ask how to sample out 50 instances from 150 instances of iris data by using Jackknife.
Is it possible?
Thanks in advance


texts: ARIMA: How to understand performance of the model?

texts:
I am new to use of ARIMA model and after working on it for a couple of days and doing research - I'm not sure how to interpret the performance of my model...
Here is what the .head() of my dataframe looks like:

As you can see it has date at its index with 4 observations/month starting at 06/01/2017. This df has 138 populated _COST observations or 155 observations with empty ones for future forecasting.
Now, moving into testing/understanding my df, here is how my rolling mean, rolling std and residual error plots look:

Here, I can confirm a couple of things: it has upwards trend, no seasonality, variance doesn't change much, residual errors are close to 0.
Finally, after doing all that - I move on to see my ARIMA model forecast...
I decided to use (1,1,0) or differenced first-order autoregressive model as if it was correct choice in my mind (Please let me know if I'm wrong!)
Here is the performance of my model; I decided to forecast last 12 weeks and this is what it looks like when I zoom in:

So, looking at this visually I'm not too happy about it...Looking at this on y-axis, because I'm working with $0.## cents it pretty much hits the numbers right.
Is this a good/bad performance? Is there some metrics to look for (such as precision, accuracy, f-1 or something in that nature as used with other machine learning models?)
Please let me know! All information and help is really appreciated as I'm only starting my professional Data Science journey :)


texts: How does "linear algebraic" weight training function work?

texts:
This answer shows that linear and polynomial function weights can be trained using this matrix operation:

$w = (X^TX)^{-1}X^Ty$

Therefore, algorithms such as gradient descent are not necessary for these functions. By my understanding, gradient descent for linear regression model finds perfect derivative for each weight so that cost function is minimal.

Before asking for connections between gradient descent and equation above, let's separate the equation in smaller steps:

$X = [1,2,3,4,5,6,7,8,9,10,11,12]$

$y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35]$

Let's turn $X$ vector to matrix by adding column of 1's which will be used to train the bias value:

\begin{bmatrix}
    1 & 1 \\
    2 & 1 \\
    3 & 1 \\
    4 & 1 \\
    5 & 1 \\
    6 & 1 \\
    7 & 1 \\
    8 & 1 \\
    9 & 1 \\
    10 & 1 \\
    11 & 1 \\
    12 & 1
  \end{bmatrix}

Transpose: $X^T$:

\begin{bmatrix}
     1 & 2 & 3 & 4 & 5 & 6 & 7 & 9 & 10 & 11 & 12 \\
     1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1
  \end{bmatrix}
Matrix multiplication $X^TX$:

\begin{bmatrix}
     650 & 78 \\
     78 & 12
  \end{bmatrix}
Inverse matrix operation $(X^TX)^{-1}$:

\begin{bmatrix}
     0.00699301 & -0.04545455 \\
     -0.04545455 & 0.37878788
  \end{bmatrix}
Matrix multiplication $(X^TX)^{-1}X^T$:

array([[-0.03846154, -0.03146853, -0.02447552, -0.01748252, -0.01048951,
        -0.0034965 ,  0.0034965 ,  0.01048951,  0.01748252,  0.02447552,
         0.03146853,  0.03846154],
       [ 0.33333333,  0.28787879,  0.24242424,  0.1969697 ,  0.15151515,
         0.10606061,  0.06060606,  0.01515152, -0.03030303, -0.07575758,
        -0.12121212, -0.16666667]])

Matrix vector multiplication $(X^TX)^{-1}X^Ty$:

\begin{bmatrix}
     0.01174825 & 2.30530303
  \end{bmatrix}

This seems to be the perfect slope for minimizing cost. But I'm unable to understand how does it exactly work.

It seems like that the algorithm such as gradient descent would've took extremely large amount of iterations to estimate perfect slope and bias value.

How/why does this equation exactly work? Is it somehow related to differentiation? Could it be compared to algorithms such as gradient descent? If so, how? Is it possible to use this equation with sigmoid functions?


texts:
I have a biased set of samples going into a binary classification sklearn pipeline, white and black samples.

It is easy for me to fetch as many black samples as required, while whites are a bit harder to get.
I suspect the black set is unbalanced and there for I have an incentive to preprocess as many black samples as my hardware permits, and pick the most representative samples.

The feature set (after dropping completely invariant features) is about 28k features.

The best approach I came up with is to cluster my black samples to a percentage of my white samples. For example say I have 500k white samples, I would then cluster the black samples to 100k clusters and pick five samples (randomly?) out of each cluster.

My questions are the following:

How can I have a transformer that will cluster the samples and mask out the unneeded samples in sklearn? Is there something ready for such tasks or should I build my own transformer based on the clustering algorithm?
Can you point at any potential issue with what I plan to do? I think one potential risk here is not clustering the white samples, but since I'm not getting enough of those anyway...
What clustering algorithms might be a good fit? I'm looking a different clustering algorithms that accept the number of clusters as parameter right now I plan to experiment with most and see which ones work best, but a direction will be appreciated.
Additionally, I'd like to read suggestions about other approaches to sample subsampling in my case.


texts: Which is the "most properly working" Bert-Ner repository

texts:
I am trying to find a repository in Github to get a Pytorch-reimplementation of the Bert model for NER task. So far, I found the following repos:

https://github.com/kamalkraj/BERT-NER
https://github.com/Louis-udm/NER_BERT_CRF
https://github.com/sberbank-ai/ner-bert

They are not exactly the same models (for example some use CRF as additionally), but for me the correctness is more important. When I check the issues, I saw that people were complaining about either mistakes in the evaluation or training procedures. So, my question, has anyone tried any of these repositories for Ner tasks before? If you know any other options, I'm open to hear.


texts: nltk.corpus for data science related words?

texts:
from job description I scraped from the internet, I've went through all nlp processes and I've got to place where I found:

freq = nltk.FreqDist(lemmatized_list)
most_freq_words = freq.most_common(100)

which outputs:

[('data', 179),
 ('experience', 86),
 ('work', 78),
 ('business', 71),
 ('team', 59),
 ('learn', 56),
 ('model', 49),
 ('skills', 47),
 ('science', 41),
 ('use', 41),
 ('build', 39),
 ('machine', 37),
 ('ability', 36),.....

and so on. My problem is I do not want to consider words like "experience", "work", and only consider keywords related to data science. I'm guessing there is a corpus for data science terms which I can use like how I use stop word corpus to not select them. Let me know if there is a way, Thanks!


texts: Are mainstream pre-trained models useful as discriminators?

texts:
In the context of GANs I see many papers designing new discriminator networks.

I'm curious about the usefulness of designing discriminators as modified versions of mainstream models like Inception, MobileNet, EfficientNet etc. My intuition is that the mentioned image classification models are way more refined and standardised than a custom discriminator for a specific paper. In addition, I think their pretrained weights should be helpful.

And if they aren't useful, I'm curious why not. Any links about this topic are much appreciated.


texts:
I want to visualize a data set for EDA having 8 parameters and 2 class labels.
I am confused which parameters to select and on what basis? Also is I want to know whether it would be better to apply dimensionality reduction techniques before visualization?


texts: What are some methodologies for performing feature selection for simple feed-forward neural networks?

texts: In multiple linear regression there is an F-test which can be used to evaluate whether or not a covariate has a meaningful impact on a model. This is typically done through either a forward selection or backwards selection algorithm. Does such a meaningful process exist for neural networks as well? The only reason I ask is because working with neural nets is an inherently stochastic process, so I do not know how I should try and get accurate bounds for the F-statistic.


texts: Including the dependent variable in your data to perform principal component analysis?

texts:
Let's say you have a data set with GPA (dependent variable) and Amount of alcohol, Amount of study, IQ, and SAT score as the independent variables. And you want to perform the principal component analysis in R for dimension reduction.

In the csv file you read in R, do you have to have GPA or do you have to remove it?


texts:
I found the definition:
Bagging is to use the same training for every predictor, but to train them on
different random subsets of the training set.
When sampling is performed with replacement, this method is
called bagging (short for bootstrap aggregating).
When sampling is performed without replacement, it is called pasting.

What is "replacement" in this context?


texts: Explain Binary Classification with output 0.5 (True)

texts:
What is the interpretation of output 0.5 of a typical classifier?

I made a prediction and the probability of that data point being from the True class is 0.5.


texts:
Calculating similarity between two users is rather straightforward.

Consider following example:

User A = {7,3,2,4,1}
User B = {4,1,9,7,5}

Products in common = {1,4,7}
Union of products = {1,2,3,4,5,7,9}

Hence the Jaccard similarity: 3/7 = 0.429

However it is not clear to me how to calculate similarity between two products.
Let's say I want to calculate similarity between products 7 and 1 from previous example, how can one achieve that?


texts: how to calculate number of datapoints within a given time interval?

texts:
I have a dataframe with one column which is a timestamp. I have converted that column to a datetime object using pandas to_datetime() method. However what I want is to count, for each time stamp, the number of timestamps that fall between this timestamp and prior 15 minutes.

  For example: If a timestamp is (2018,7,6,13,55) [year,month, day,
  hour, minute format], then I want to count the number of timestamps
  (or datapoints) that fall between this timestamp and the timestamp
  (2018,7,6,13,40).

It is easy to see what needs to be done, but it involves repeatedly going back and forth over the dataset. I have close to 750,000 datapoints and what I am thinking is not efficient.

How can I do it efficiently?

Any suggestion/hint would be highly appreciated.

Thanks.


texts: Fully endogenous models for predicting multivariate time series

texts:
I have a formal social science background but I am new to data science. My interest is in building predictive models for applications in the social sciences, mostly (but not only) in economics.

I am interested in the following kind of setups:

I have data that describe the evolution of a number of variables $j \in J$ for a number of "individuals" $i\in N$ across time periods $t \in \{1\dots, T\}$.
For example, "individuals" $i\in N$ could be countries, with $j=1$ being GDP, $j=2$ inflation, $j=3$ interest rate, $j=4$ unemployment rate, etc, although in practice I would look at situations where $\#N > 195$ (maybe $i\in N$ would be regions, or counties, rather than countries).
As the above example suggests, I am interested in situations where all my variables of interest are likely to be related to one another.
I am looking for models that, based on data for $t\in \{1,\dots, T\}$ can forecast the co-movement of all my variables of interest for $t > T$ (at least a couple of periods ahead). In particular, I don't want to have to assume some particular future scenario for all but one variable in order to get a prediction about the last one.

I am interested in prediction only, not in satistically interpreting my model.

Being trained as an economist, I know that Vector Autoregression (VAR) is one option for such "fully endogeneous" models. I have tried to learn a little more about forecasting and time-series (e.g., read through https://otexts.org/fpp2) but found little alternatives to VAR so far. Now, VAR might be fine and it might be the only option out there. But I would like to know if there are alternatives modeling techniques one may want to consider for these kinds of problems.

My questions:

What would be some alternative to VAR be to tackle these kinds of endogenous prediction problems (if any)?
Are there any resources where the application of these alternative techniques would be discussed specifically in this context?


texts: Removing strings after a certain character in a given text

texts:
I have a dataset like the one below. I would like to remove all characters after the character ©. How can I do that in R?
data_clean_phrase <- c("Copyright © The Society of Geomagnetism and Earth",
"© 2013 Chinese National Committee ")

data_clean_df <- as.data.frame(data_clean_phrase)


texts: Too low accuracy on MNIST dataset using a neural network

texts:
I am beginning with deep learning. This is an implementation of a simple neural network with just 1 hidden layer on MNIST dataset. Why is it that the loss doesn't change at all after any epoch? It clearly means that it is not learning at all. The accuracy is approx. 11% that is like random guessing. But should it be so less?
I have used Adam optimizer and cross_entropy loss.

input_nodes = 784
hl1_nodes = 64
output_nodes = 1

from keras.datasets import mnist
(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train_reshaped = X_train.reshape(X_train.shape[0],784)

model = Sequential()
model.add(Dense(hl1_nodes, activation='relu', input_shape=(input_nodes,)))
model.add(Dense(output_nodes, activation = 'sigmoid'))

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

history = model.fit(x=X_train_reshaped, y=y_train, validation_split=0.33, verbose=1,epochs=10)

#output
Train on 40199 samples, validate on 19801 samples
Epoch 1/10
40199/40199 [==============================] - 4s 87us/step - loss: -55.0254 - acc: 0.1142 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 2/10
40199/40199 [==============================] - 3s 76us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 3/10
40199/40199 [==============================] - 3s 74us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 4/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 5/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 6/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 7/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 8/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 9/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 10/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088

What am I missing?
Edit: It is same upto the 4th digit even after 90th epoch.


texts: Predicting earthquakes using disturbances in DTH TV transmission

texts:
It is said that before an earthquake happens, a viewer experiences disturbances in DTH TV transmission in the form of distorted images on the screen which automatically correct after a few seconds.

Is it possible to identify patterns of such disturbances by continuously monitoring TV images so that earthquakes can potentially be predicted at least few minutes in advance and many lives could be saved?


texts: Feature reduction by removing certain columns in dataframe


texts: Multioutput classification in Keras - how to get multivariate probabilities and deal with unseen classes

texts:
I'm struggling to design in Keras a deep neural network for multioutput classification model. The network works in tandem with external logic in a kind of feedback loop: in each iteration the external module generates the training set, on which the network is trained and then in next iteration the network supports the module in another round of training set generation. I'm rather at rookie+ level and I have encountered 2 problems I do not know how to deal with:

how to make a network predict vectors that will not show up in training set, but still account for underlying correlations between vector components
how to yield a multivariate distribution on multioutput space.

To be more specific, the "ground truth" of the model is $N$-length vector of binary labels: $y \in \{ 0, 1\}^{N} $. This represents some yes/no $N$-dimensional decision and is generated by the external module. The key issue is the yes/no components are highly correlated, $N$ maybe large and the ground truth is not "deterministic" - i.e. it's just some approximation that is expected to get better over the feedback learning process described above.

So far I have designed a simple classification network based on Keras sequence model with $N$ output layer units.

Restating my problems in more specific way:

ad 1) there are $2^{N}$ possible output vectors ("metaclasses"). As $N$ is large, almost surely not all of them will appear in my training set. But I want my network to be flexible enough to be able to come up with unseen metaclasses but still take care of high correlations between single yes/no components.

ad 2) I would like the network to output a few (say three) most probable vectors. In other words, I would like it to yield a (obviously some cut-off) probability distribution over vectors.


texts: Good books for Hadoop, Spark, and Spark Streaming

texts:
Can anyone suggest any good books to learn hadoop and map reduce basics?

Also something for Spark, and Spark Streaming?

Thanks


texts: Which algorithm to use to match two categories with n dimensions

texts:
Suppose I have examples of two categories, Products, and Aisles.

Each product has variables like price, weight, moisture rating etc and then Aisles has variables like max weight it can handle, moisture near it, etc.

Now I want to run some algorithm which classifies which products should go in which aisles based on product and aisles variables.

examples if product A has weight 10kg, moisture it can handle x, it should go to Aisle which can handle that weight and has that moisture rating.

Provided I have same variables across product and aisles, how can I solve my problem?


texts: In Incremental Learning will the model be updated automatically?

texts:
I came across Incremental Learning algorithms paper, where incremental algorithms are compared. I have problem with general understanding. Will the model be updated /adapts itself automatically when new data comes in?

Does it know by itself that new  data has arrived and it learns?

In general, can anyone explain how training, testing, and model adaption is carried out with such incremental algorithms?


texts: Understanding how many biases are there in a neural network?

texts:
I am trying to understand biases in neural nets, but different websites show very different answers.

For example, how many biases is there in a fully connected neural network with a single input layer with 5 units and a single output layer with 4 units? And what about a fully connected neural network with a single input layer with 5 units, a single hidden layer with 4 units, and a single output layer with 3 units?

For example, if I understand this correctly, https://ai.stackexchange.com/questions/17584/why-does-the-bias-need-to-be-a-vector-in-a-neural-network, the answer of the first should be 5 and for the second 4 + 3. Each neuron except for in the input-layer has a bias.

However, at https://ayearofai.com/rohan-5-what-are-bias-units-828d942b4f52, it is explained such that each layer including the input-layer has one bias. So the answer to the example above is one in the first and two in the second.

What is correct? What am I misunderstanding here?


texts: how to balance the data set with different number of observations

texts:
I am pretty new to ds field and recently I was working on a self project of the clustering model. My goal is to create clusters and see in each cluster what the common features are between customers. The data set contains customer's information, when and what product they order and more. Below is a sample of the data:

id   gender   age   order_dt   ship_dt  product
1     male     23   1/2/2018   1/9/2018    a
1     male     23   1/5/2018   1/6/2018    b
2    female    45   1/10/2018  1/20/2018   c
3    female    30   1/1/2018   1/2/2018    a
3    female    30   1/15/2018  1/20/2018   c
3    female    30   1/21/2018  1/21/2018   b
3    female    30   1/29/2018  2/1/2018    a

However, each id could contribute to a different number of records in the data set. Some could have a lot of records because they order many times while some only order once. I have googled around for unbalanced data, but most of them are talking about unbalancing in one category(feature maybe?) instead of the unbalanced number of observation.
Should I aggregate on each id so that each id would only have 1 record in the data set or if there is any technique to handle data like this?

Thanks in advance,


texts: How can I use data science / machine learning / statistics / data analytics techniques to find better customers in the future?

texts: I work for a business to business company with a data set containing approximately 2700 businesses as customers. For each business customer I have approx number of staff, location, industry, the sales rep managing their account and for the last 7 years how much profit I have made with each of them for each month. What data science/machine-learning/statistics techniques or analysis can I perform to use this data to select which businesses I should try to acquire as customers in the future? And what if any methods might be worth considering to enrich this data set?


texts:
I have a series of seemingly random data dripping in one value at a time through time. Although it appears to be random, the data forms clusters when certain attributes are analysed which the charts show. I'm trying to avoid the fallacy of seeing patterns that are not there, but across several datasets the data does indeed appear to form clusters through time. I'm interested in trying to:

detect the clusters of either the blue squares and / or the white squares in the data I have
ignoring data which are outliers
predicting the future shape of the clusters as data is dripped in one column at a time. (There appears to be a certain momentum through time of the cluster shapes where the current cluster areas of the new data is in nearby region of the preceding clusters)

They appear to take on regular semi-predictable shapes which (to my eyes at least) follow a momentum from the data before. How could I detect this?

My questions are:

is this cluster analysis, and
what would be the best way to detect the white squares and / or the blue areas. I'm guessing the blue areas would be the easiest to cluster?
the white squares appear fairly close to the previous y-values and may continue at a certain angle up or down in value. How is the best way to predict this?


texts: When is feature transformation required?

texts:
I was fitting machine learning models to clean data(Imputed missing values, removed unnecessary features etc). I didn't transform the features that are skewed. Before moving forward, I want to understand how important feature transformation is to fit data into a model. Any opinions?

(I know what happens in Random Forest, but unable to comprehend for other ML models)


texts: Which metric to use for classify a binary logistic regression

texts:
I still novice in machine learning. My problem is as follow :

I use two binary logistic regression models for making comparaison. Additionally to LogLoss and accuracy, I would like to use for metric : precision / recall/ F1-mesure.

I have used the formulat :

print (classification_report(y_test, y_pred.round()))

My question is: which one, for the tree metric value, should I used. is it the weighted average ore the parametres of the true values (number 1 in picture)


texts: Differences between normalization and standarization in multiple regression

texts:
Consider the following question regarding multiple regression

1) Can someone explain why we have to transform dependent variable using log-transformation (Normalization) when appear positive skewed y variable in regression model?

2) After log-transformation whether do I need to standardize that y variable using min max scale or StandardScaler methods?

3) If independent variables also have skewed data then do I need to Normalization for that variables or standardizing is enough?


texts: recurrent weights on LSTM Data-gate? (the one with tanh)

texts:
Looking at the literature, there are 2 distinct approaches to LSTM

Some people use recurrent weights with Input, Forget, Output - notice, their equations don't even mention dataGate, they start from describing the $f$ or $i$ gate (1), Wikipedia: (2)

Lke this:

Other people use recurrent weights with dataGate (z), Input, Forget, Output (1) (2)  (3), (4)

Like this:

Having tried the first approach (I still used standard weights on dataGate but no recurrent weights in that place) seems fine but a little odd - network converges well, but very rarely jumps over local mimima, as if too high learning rate.

Personally I like the second approach (recurrent weights on all 4 entry points), which one do I use?

Edit:

to make things worse, the "LSTM Peephole paper", page 121 describes 3 connections (no peephole for dataGate).

This destroys the equal-sizes of my matrices, because now 3 gates use the Usual, Recurrent and Peephole weights, but the very first gate only uses the Usual weights


texts: Implementing U-Net segmentation model without padding

texts:
I'm trying to implement the U-Net CNN as per the published paper here.
I've followed the paper architecture as closely as possible but I'm hitting an error when trying to carry out the first concatenation:

From the diagram, it appears the 8th Conv2D should be merged with result of the 1st UpSampling2D operation, however the Concatenate() operation throws an exception that the shapes don't match:
def model(image_size = (572, 572) + (1,)):

    # Input / Output layers
    input_layer = Input(shape=(image_size), 32)

    """ Begin Downsampling """

    # Block 1
    conv_1 = Conv2D(64, 3, activation = 'relu')(input_layer)
    conv_2 = Conv2D(64, 3, activation = 'relu')(conv_1)

    max_pool_1 = MaxPool2D(strides=2)(conv_2)

    # Block 2
    conv_3 = Conv2D(128, 3, activation = 'relu')(max_pool_1)
    conv_4 = Conv2D(128, 3, activation = 'relu')(conv_3)

    max_pool_2 = MaxPool2D(strides=2)(conv_4)

    # Block 3
    conv_5 = Conv2D(256, 3, activation = 'relu')(max_pool_2)
    conv_6 = Conv2D(256, 3, activation = 'relu')(conv_5)

    max_pool_3 = MaxPool2D(strides=2)(conv_6)

    # Block 4
    conv_7 = Conv2D(512, 3, activation = 'relu')(max_pool_3)
    conv_8 = Conv2D(512, 3, activation = 'relu')(conv_7)

    max_pool_4 = MaxPool2D(strides=2)(conv_8)

    """ Begin Upsampling """

    # Block 5
    conv_9 = Conv2D(1024, 3, activation = 'relu')(max_pool_4)
    conv_10 = Conv2D(1024, 3, activation = 'relu')(conv_9)

    upsample_1 = UpSampling2D()(conv_10)

    # Connect layers
    merge_1 = Concatenate()([conv_8, upsample_1])

Error:
Exception has occurred: ValueError
A `Concatenate` layer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(32, 64, 64, 512), (32, 56, 56, 1024)]

Note that the values 64 and 56 correctly line up with the architecture.

I don't understand how to implement the model as it is in the paper. If I change my code to accept an image of shape (256, 256) and add padding='same' to the Conv2D layers, the code works as the sizes are aligned.
This seems to go against what the authors specifically state in their implementation:

Could somebody point me in the right direction on the correct implementation of this model?


texts: Transform test data when using a persistent model

texts:
I'm quite new to data science and only slowly following the necessary steps to get valid results using scikit-learn. As far as I understand you fit and transform the training data and only transform the test data (using the parameters retrieved by the earlier fitting). For my project a persistent model is necessary, for that I export the trained model using joblib.

When applying the model on test data later, is there a way to retrieve the parameters (for transformation) generated during the training process?


texts: I am creating a model using xgboost. Regarding its parameters, its objective is survival:cox and its eval_metric is cox-nloglik. The output Y ranges from -800 to 800. However, the predicted values are way to large (in range from 10^3 to 10^13). Why is this? What is the outcome of a Cox regression in xgboost?


texts: Text classification with Word2Vec on a larger corpus

texts:
I am working on a small project and I would like to use the word2vec technique as a text representation  method. I need to classify patents but I have only a few of them labelled and to increase the performance of my ML model, I would like to increase the corpus/vocabulary of my model by using a large amount of patents. The question is, once I have train my word embedding feature, how to use this larger corpus with my training data - my labelled data?
My data set is composed by 2000 patents which are labelled.
The patents used to train my word embedding corpus are 3 millions (some of my 2000 labelled patents are already included in this larger corpus) which I trained using Gensim.
Do you have any suggestions on how to do it?
Thank you very much in advance.
Rob


texts: Feeding XLM-R embeddings to neural machine translation?

texts:
I’m very new to the field of deep learning. My aim is to make a translation between Catalan to Catalan Sign Language. The grammar of the two languages is different

  Input: He sells food.
  Output (sign language sentence): Food he sells.

I've been playing around with XLM-R and go the token id like this

input Ids: [200, 100, 2003, 1037, 3835, 3351, 5012, 300, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

I don't know how to use the embeddings in Sequence to Sequence NMT model. or any other means to do machine translation with a very small data set. The language is low resource language

import torch
from transformers import XLMRobertaModel, XLMRobertaTokenizer

tokenizer = XLMRobertaTokenizer.from_pretrained('xlm-roberta-large')
model = XLMRobertaModel.from_pretrained('xlm-roberta-large')

def get_ids(tokens, tokenizer, max_seq_length):
token_ids = tokenizer.convert_tokens_to_ids(tokens)
input_ids = token_ids + [0] * (max_seq_length - len(token_ids))
return input_ids

s = "test sentence"
stokens = tokenizer.tokenize(s)
print(stokens)
stokens = ["[CLS]"] + stokens + ["[SEP]"]
input_ids = get_ids(stokens, tokenizer, 15)

print(tokenizer.convert_tokens_to_ids(['test']))
print(tokenizer.convert_tokens_to_ids(['▁test']))
print(tokenizer.convert_ids_to_tokens([26130]))
print(tokenizer.convert_ids_to_tokens([30521]))
tokens_tensor = torch.tensor([input_ids])
print(input_ids)
print(tokens_tensor)
```


texts: DONUT- Anomaly detection Algorithm ignores the relationship between sliding windows?

texts:
I'm trying to understand the paper : https://netman.aiops.org/wp-content/uploads/2018/05/PID5338621.pdf about Robust and Rapid Clustering of KPIs for Large-Scale Anomaly Detection.

Clustering is done using ROCKA algorithm.

Steps:

1.) Preprocessing is conducted on the raw KPI data to remove amplitude differences and standardize data.

2.) In baseline extraction step, we reduce noises, remove the extreme values (which are likely anomalies), and extract underlying shapes, referred to as baselines, of KPIs. It's done by applying moving average with a small sliding window.

3.) Clustering is then conducted on the baselines of sampled KPIs, with robustness against phase shifts and noises.

4.) Finally, we calculate the centroid of each cluster, then assign the unlabeled KPIs by their distances to these centroids.

I understand ROCKA mechanism.

Now, i'm trying to understand DONUT algorithm which is applied for "Anomaly Detection".

How it works is :

DONUT applies sliding windows over the KPI to get short series x and tries to recognize what normal patterns x follows. The indicator is then calculated by the difference between reconstructed normal patterns and x to show the severity of anomalies.
In practice, a threshold should be selected for each KPI. A data point with an indicator value larger than the threshold is regarded as an anomaly.

Now my question is :

IT seems like DONUT is not robust enough against time information related anomalies. Meaning that it works on a set of sliding windows and it ignores the relationship between windows. So the window becomes a very critical parameter here. So it might generate high false positives. What I'm understanding wrong here?

Please help and make me understand how DONUT will capture the relationship between sliding windows.


texts: Help required to implement the below model using Bi-GRU

texts:
As you can see in above images I need to model Bi-GRUs stacked as shown in table which takes input (N,1,64) and outputs (N,204). The input data is binary number stream and so is output data. Can anyone please help me get started?

Thank you.


texts: Attention using Context Vector: Hierarchical Attention Networks for Document Classification

texts:
In the paper, "Hierarchical Attention Networks for Document Classification", the authors use attention to compose words to sentences and then sentences to a document representation. They make use of a context vector $u_w$ to compute the attention weights for the annotation of each word in the sentence.

The paper states in section 2.2, "The context vector $u_w$ can be seen as a high level representation of a fixed query “what is the informative word” over the words. The word context vector $u_w$ is randomly initialized and jointly learned during the training process."

This implies that the context vector is independent of the sentence input. It remains the same for every sentence and is learned as a parameter of the neural network. If this is the case, how will $u_w$ accurately provide attention weights for words in a random sentence, given that sentences are so diverse in meaning.

I do not understand the workings of $u_w$, since it is independent of the sentence input.

Can someone explain?


texts: I want to replace XGBRegressor with a simple model to make feature selection

texts: I will make some for loop on to select the best features by my Data frame is big 10M row and about 50 columns so if i replaced xgb with a single Decision tree would it be the same results for the best features and i can choose them on xgb or it may differ, i just want to save time as much as possible.


texts:
From the definition of resent from wikipedia: it is mentioned that resent model uses fewer layers in the initial training stages. This speeds learning by reducing the impact of vanishing gradients, as there are fewer layers to propagate through. The network then gradually restores the skipped layers as it learns the feature space.

I am new to this domain, and have tried using resent50 and resnet18 for toy examples, but I am not sure about these theoretically parts. Can someone help to answer or give directions to study material for below queries:

How does resnet model restores the skipped layers as it learns the feature space?
it seems like the certain layers are skipped due to problem of vanishing gradients, but why does not the adjacent layers (who's activation is used while skipping certain layers) face the same issue?


texts: Orange, K-means clustering, memory error

texts:
I have a dataset of over 67 000 records, and I'm trying to run a k-means cluster analysis on that. Orange returns a memory error.
The data is in an excel file, but I also tried to load it from a csv. Omitting some records and downsizing it under 65k records also couldnt help.
Hierarchical clustering also doesnt work for the same reason.

Does anyone have any ideas on what to do here?

Error code:

  MemoryError
  Traceback (most recent
  call last):
  File
  "C:\Python34\lib\site-packages\Orange\widgets\gui.py", line 2220, in

      b.button = btn = button(b, master, label, callback=lambda: do_commit())   File
  "C:\Python34\lib\site-packages\Orange\widgets\gui.py", line 2191, in
  do_commit
      commit()
  File "C:\Python34\lib\site-packages\Orange\widgets\unsupervised\owkmeans.py",
  line 246, in run
      self.run_optimization()
  File "C:\Python34\lib\site-packages\Orange\widgets\unsupervised\owkmeans.py",
  line 225, in run_optimization
      self.optimization_runs.append((k, kmeans(self.data)))
  File "C:\Python34\lib\site-packages\Orange\projection\base.py", line 28, in
  call
      clf = self.fit(data.X, data.Y)
  File "C:\Python34\lib\site-packages\Orange\clustering\kmeans.py", line 23,
  in fit
      proj = proj.fit(X, Y)
  File "C:\Python34\lib\site-packages\sklearn\cluster\k_means_.py", line 821,
  in fit
      n_jobs=self.n_jobs)
  File "C:\Python34\lib\site-packages\sklearn\cluster\k_means_.py", line 277,
  in k_means
      tol = _tolerance(X, tol)
  File "C:\Python34\lib\site-packages\sklearn\cluster\k_means_.py", line 162,
  in _tolerance
      variances = np.var(X, axis=0)
  File "C:\Python34\lib\site-packages\numpy\core\fromnumeric.py", line 2938,
  in var
      keepdims=keepdims)
  File "C:\Python34\lib\site-packages\numpy\core_methods.py", line 102, in
  _var
      x = asanyarray(arr - arrmean)
  MemoryError


texts: How to "count" certain events in a time series

texts:
What techniques are suitable and what do I need to learn in order to
detect and count the number of "events" (pic) that consist of:

a) shape 1 3 and 4 and not of (2 and 5) or

b) of all shapes above the blue line with a certain minimum "volume" like shapes 1 and 4?

My first idea is to use a window function and somehow integrate the area but i want to be sure to use an appropriate procedure. I havent worked with time series before and I dont know were to get startet. I have to use python for this project.


texts:
In the first problem, it's been told to accept the maximum number of good customers, if at least 98% of the customers that are do not repay their debt correctly identified.
I am confused about what is meant by this? Do I need to train the models and set the threshold value accordingly so that I may obtain 98% accuracy? And for the second one, I need to decrease the threshold value so that accuracy comes down to 85%?
I am stuck in this for a while. Please help me show the correct path. Thanks.
data: German Credit Risk data.
Variables: Several independent variables with the dependent variable "Credit_Risk" which responses in "Good" and "Bad".


texts: Dueling Network gradient with respect to Advantage stream

texts:
Looking at Dueling DQN:

$Q = V + A - mean(A)$

For simplicity, let's assume we are working with 4 neurons. Recall that Value stream only has 1 neuron $(v_0)$

Re-writing the above equation, we get:

$$
 \left[  \begin{array}{c}
 Q_0\\Q_1\\Q_2\\Q_3
\end{array}
\right] =
\left[  \begin{array}{c}
 v_0\\v_0\\v_0\\v_0
\end{array}
\right] +
 \left[  \begin{array}{c}
 a_0\\a_1\\a_2\\a_3
\end{array}
\right] -
 \left[  \begin{array}{c}
 \frac{1}{N}(a_0 + a_1 + a_2 + a_3)
\\\frac{1}{N}(a_0 + a_1 + a_2 + a_3)
\\\frac{1}{N}(a_0 + a_1 + a_2 + a_3)
\\\frac{1}{N}(a_0 + a_1 + a_2 + a_3)
\end{array}
\right]
$$

Question:

V (the Value) is a usual Dense Layer which has 1 neuron at the output.

A (the Advantage) is a usual Dense Layer which has N neurons at the output - what is the correct gradient vector to be passed to the neurons of this Advantage stream?

Thoughts:

From this post:

  Since Q is a simple sum of functions you have:

  $$\nabla_{\theta} Q(s,a) = \nabla_{\theta}V(s) + \nabla{\theta}A(s,a)
 - \frac{1}{numActions} \sum_{a'}\nabla_{\theta}A(s, a')$$

  You get the gradients of the V and A networks as usual by backprop.

To me, the correct gradient vector to be passed to the advantage stream is:

$$gradForA = \frac{dE}{dQ}\frac{dQ}{dA}$$

however, the second fraction of the equation is what makes me puzzled.
Is it as simple as:

$$\frac{dQ}{dA} =  \left[  \begin{array}{c}
1-\frac{1}{N}\\1-\frac{1}{N}\\1-\frac{1}{N}\\1-\frac{1}{N}
\end{array}
\right] $$

It's probably not, especially if we look at the "mean" vector, in the example above.
We can see that its every entry contains contribution of all advantage neurons - because they are summed.

Am I doomed to perform 2 backpropagations for the advantage dense layer, in parallel? - One for $+A$ and one for $-mean(A)$.  I would then add-up (component-wise) the two gradient vectors. Sounds like a clumsy idea..


texts: I am new to tensorflow. I have manged to train and validate a CNN, saved the session through the Saver object into a CPKT file and loaded it back. Now I'd like to use the trained model in order to check how it performs against a photo I took myself. I can't figure out how to do it.


texts: Data augmentation: ImageDataGenerator vs openCV

texts:
I would like to increase the data in my dataset to create a CNN deep learning classification model.

Which is better for the model, using data augmentation by ImageDataGenerator or using openCV to increase the data?

By the way, I am using Keras and floydhub.


texts: Taking Neural Network's false positives as the recommendation system result?

texts:
I am creating a recommendation system and considering two parallel ways of formalizing the problem. One classical, using proximity (recommend the product to the customer if a majority vote of 2k+1 customers closest by has the product), and another one that I have trouble understanding but seems valid to some extent.

The approach I'm thinking about is:

1) Fit a highly regularized neural network (to make sure it doesn't overfit the training set) for a classification task that can predict if the person does or doesn't have given product

2) Make sure test accuracy is as close to train accuracy as possible

3) Take false positives (customers who don't have the product originally but the NN predicted that they have it) predicted on the whole dataset (the training set as well) as the result - the people I should recommend the product to

Now, I am aware of why in general one wouldn't want to take that approach but I also can't exactly explain why it wouldn't return people 'close by' to each other that 'should' have given product in a similar sense like the KNN-based approach. I'm not sure how to analyse this problem exactly to validate, modify or reject the idea altogether.


texts: considerations on growing number of parameters in a neural network

texts:
I have built a convolutional neural network, in particular an AlexNet, and I have noticed that the number of parameters grow a lot as we go forward in the network. Now, I know that the parameters in the network is what we want to learn, and also know that its number grows as we go forward in the network because a summation wth other neurons is performed.

To me it seems like a crucial point to learn to unerstand truly how neural networks work, so my question is:

What are the implications of this growing number of parameters (if any)? And what considerations is it possible to do on this fact?

Thanks in advance.

[EDIT]For exampple, if I use an AlexNet, I have the following table:

which goes on until the last layer. The parameters in this table should be the weights.


texts: How to model this "un predicatability" problem?

texts:
Imagine modeling the "input(plaintext) - output(ciphertext)" pairs of an encryption algorithm as a data science problem. Very informally, the strength of an encryption scheme is measured by the randomness (unpredictability, or entropy) of the output. This is counter-intuitive to classic regression problems, which are frequently used for prediction. Say, informally, the strength of an encryption scheme is determined by the number of such input-output pairs needed beyond which it becomes predictable.

How do we model this as data science problem? Given all pairs of two different encryption schemes, can we determine which is stronger just by using the input-output pairs of both the schemes? Is there any other way apart from regression to solve this problem?


texts: Is it reasonable that a problem "solved" by a traditional ANN can also be solved by a CNN?

texts:
There is a data science numerical problem, which me and my team were able to get an ANN model that predicts down to a 1% MAPE error (with roughly 70000+ trainable parameters).

Given the nature of the dataset, this data can also be processed as images, which has led me to trying a CNN based approach to solve the same. As of now, I'm able to converge to error figures of 14% with around 2000 parameters. Is it reasonable to assume that I'd be able to bring down the error even further with a CNN based architecture (without a significant increase in the parameters)?

EDIT: Basically, what I mean by image is, in the ANN case, the data is being processed as a 1D vector, but as a 2D vector in the CNN case.


texts: Is it possible to change pandas column data type within a sklearn pipeline?

texts:
Sklearn pipeline I am using has multiple transformers but one of the initial transformers returns numerical type and the consecutive one takes object type variables.

Basically I need squeeze in a:

data[col] = data[col].astype(object)

for the required columns within the pipeline.

Is there any way to do it?

Note: I am using Feature-engine transformers.


texts: what do the percentages in the leaves of a decision tree represent?

texts:
in figure B), there are leaves (gray boxes) with 3 values, for example, the leftmost leaf has 19.3 (28/8.7%) as its values, the 3 values are (19.3, 28, and 8.7%).

19.3 is the average value of the instances that reach that leaf
28 is the number of instances (out of the 209 total instances)
what does the 8.7% mean?

thanks

!]1


texts: Predicting car failures with machine learning

texts:
I want to start with machine learning with a small prediction problem but I'm not sure I chose the right approach. I want to make a program that gets data of mechanical failures on cars (manufactured time, failure time, reason, and different characteristics of the car). Then I would give the data of new cars that will be released to market and I would try to predict when would they fail.

I was reading that the best approach is using survival analysis with R but since I'm not really familiar with this algorithm, I was wondering If there's any other approach.


texts:
I have sparsely populated matrix of users as rows  with columns being categorical answers to various questions ( question are of various domain about  preferences / behaviors of the users ) . answers may be be either numerical ( for example an answer to the question "what's the number of children in your household ? " ), or categorical   ( "specify your education level  ? BS / PHd  / etc."). As mentioned the matrix is sparse , and the aim is to infer the missing entries.

do you think matrix factorization techniques ( for example  ALS ) could be suitable for solving this ? ( with proper normalization of the response ) and do you suggest another learning algorithm ?


texts: Word2Vec: Identifying many-to-one relationships between words

texts:
Standard introductory examples in Word2Vec, like king - queen = man - woman and tokyo - japan = london - uk, involve one-to-one relationships between words: Tokyo is the exclusive capital of Japan.
More generally, we might want to test for many-to-one relationships: e.g. we might want to ask if Kyoto is a city in Japan. I presume we are still interested in vectors of the form kyoto - japan, houston - us, etc., but these vectors are no longer equal.
Do these "relationship vectors" form a particularly interesting vector space? Do they sample some known distribution? How can I check a many-to-one relationship from the word embeddings?


texts: Implement Sliding Window Algorithm to Detect Spikes

texts: How do I implement sliding window algorithm with a window size of 10 and visualize the data iteratively to see spikes/possible outliers in the dataframe, using python? Please help a beginner.


texts: How are Decision Trees averaged in Random Forest?

texts:
We all know that a Random Forest is an ensemble of Decision Trees, whose results are averaged.

Every source I find simply talks about "averaging trees", but how does this "averaging of trees" happens? Can anyone exaplain it in detail?


texts: Cluster algorithm which minimizes a distance while fulfilling a constraint

texts:
I have data of 90 persons.

Data of a person:
- x coordinate
- y coordinate
- score (1 to 6)

I want to form groups of 9 people each so that:
- the distance between people in one group is as small as possible
- people with the same coordinates are in different groups
- people within a group should have different scores

I am not expecting a full-fledged algorithm but rather hints in the right direction. So far I am using a simple k-means algorithm which forms groups so that the distance between people gets minimized however the two other constraints are not considered, unfortunately.


texts: Exploratory Data Analysis and selecting good predictor variables ?

texts: In what way would exploratory data analysis aid in feature selection, other than to preprocess the data ? Say, if a bivariate analysis was conducted for each predictor variable w.r.t. the target variable, in what way would this help with feature selection, if possible ?


texts: How to classify a set of words into one of the given labels

texts:
I have three labels: amusement, calm and energetic.
I get sets of words like:
Set1 = {Cloud
Sky
People in nature
Plant
Flash photography
Happy
Shorts
Grass
Leisure
Recreation}
Set2 = {Plant
Green
Natural landscape
Natural environment
Branch
Tree
People in nature
Shade
Wood
Deciduous}
I want to classify these group of words into one of the labels. What do you guys think? Set1 should be labelled energetic and Set2 should be labelled calm.


texts: Will my validation loss eventually go down?

texts:
I'm currently training a binary classifier that takes in 2 inputs, and outputs which object it thinks is "better."
I have an absolutely massive dataset, about 2 trillion records, and I'm feeding these records into my network about 300k records at a time. Overfitting isn't really a concern as I'm only running one epoch, so the network is only really seeing new data every cycle.
So far, the training loss is decreasing steadily, and my training accuracy is rising.
My validation loss is decreasing slightly, but fluctuating a lot. My validation accuracy is rising at about the same rate as my training accuracy is.
Will I eventually see a large drop in validation loss as the network gets more "confident?" I know that's a very hard question to answer without knowing more details, but have you seen this type of behavior before in your models? I'm just a little spooked since these experiments take up so much time to run.


texts: The string "mean_squared_error" appears to be deprecated in cross_val_score now, and it's saying to use neg_mean_squared_error. Is this metric literally just the negative of the MSE?


texts: How to apply oversampling when doing Leave-One-Group-Out cross validation?

texts:
I am working on an imbalanced data for classification and I tried to use SMOTE previously to oversampling the training data. However, this time I think I need to use a leave-on group out (LOGO) cross-validation because I want to leave one subject out on each CV.

I am not sure if I can explain it nicely, but, as my understanding, to do k-fold CV using SMOTE we can loop the SMOTE on every fold, as I saw in this code on another post. Below is an example of SMOTE implementation on the k-fold CV.

from sklearn.model_selection import KFold
from imblearn.over_sampling import SMOTE
from sklearn.metrics import f1_score

kf = KFold(n_splits=5)

for fold, (train_index, test_index) in enumerate(kf.split(X), 1):
    X_train = X[train_index]
    y_train = y[train_index]
    X_test = X[test_index]
    y_test = y[test_index]
    sm = SMOTE()
    X_train_oversampled, y_train_oversampled = sm.fit_sample(X_train, y_train)
    model = ...  # classification model example
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    print(f'For fold {fold}:')
    print(f'Accuracy: {model.score(X_test, y_test)}')
    print(f'f-score: {f1_score(y_test, y_pred)}')

Without SMOTE, I tried to do this to do LOGO CV. But by doing this, I will be using a super imbalanced dataset.

X = X
y = np.array(df.loc[:, df.columns == 'label'])
groups = df["cow_id"].values #because I want to leave cow data with same ID on each run
logo = LeaveOneGroupOut()

logo.get_n_splits(X_std, y, groups)

cv=logo.split(X_std, y, groups)

scores=[]
for train_index, test_index in cv:
    print("Train Index: ", train_index, "\n")
    print("Test Index: ", test_index)
    X_train, X_test, y_train, y_test = X[train_index], X[test_index], y[train_index], y[test_index]
    model.fit(X_train, y_train.ravel())
    scores.append(model.score(X_test, y_test.ravel()))

My question will be:
How should I implement SMOTE inside a loop of leave one group out CV, I am confused about how to define the group list for the synthetic training data.

I'd be happy to provide more information. Thanks!


texts: Running out of memory when calculating loss using SigmoidFocalCrossEntropy

texts:
Code Versions:
python == 3.8
tensorflow == 2.2.0
tensorflow-addons == 0.11.2

Recently I've been using tensorflow addon's focal loss function for one of my models. In order to better understand/demonstrate what's going on, I've been trying to recreate some of the graphs from the original paper - specifically, figure 4, which shows a cumulative sum plot of the minority and majority loss separately using different gammas on a converged model.
def calcCSP(pred, Y_test, g, label):

    true = np.asarray(Y_test)
    idsPos = np.where(true == 1)
    idsNeg = np.where(true == 0)

    pPos = pred[idsPos]
    yPos = true[idsPos]

    pNeg = pred[idsNeg]
    yNeg = true[idsNeg]

    if label == 1:
        p = pPos
        y = yPos
        title = "Positive Loss Distribution"
    else:
        p = pNeg
        y = yNeg
        title = "Negative Loss Distribution"

    p = tf.convert_to_tensor(p)
    y = tf.cast(y, tf.float32)

    fl = tfa.losses.SigmoidFocalCrossEntropy(alpha=PARAMS['alpha'], gamma=g)

    loss = fl(y, p)

    x = np.sort(loss)

    # Normalized Data
    x = x/sum(x)

    cdf = np.cumsum(x)
    n = len(x)
    share_of_population = np.arange(1, n + 1) / n

    cdf_materials = {"shares": share_of_population,
                     "cusum": cdf}

    return cdf_materials

cdfListPos = []
cdfListNeg = []

gammas = [0, 2, 4, 6, 8]

for g in gammas:
    # will need to store all this
    cdf_matPos = calcCDF(preds, y_test, g, 1)
    cdf_matNeg = calcCDF(preds, y_test, g, 0)
    cdfListPos.append(cdf_matPos)
    cdfListNeg.append(cdf_matNeg)

posplot = plt.figure()
for i in range(len(gammas)):
    plt.plot(cdfListPos[i]['shares'], cdfListPos[i]['cusum'], label= r'$\gamma$ = ' + str(gammas[i]))
    plt.title('Positive Points CSP')
    plt.ylabel('Cumulative Normalized Loss')
    plt.legend()

negplot = plt.figure()
for i in range(len(gammas)):
    plt.plot(cdfListNeg[i]['shares'], cdfListNeg[i]['cusum'], label=r'$\gamma$ = ' + str(gammas[i]))
    plt.title('Negative Points CSP')
    plt.ylabel('Cumulative Normalized Loss')
    plt.legend()

I've managed to run my code okay on smaller datasets - however, once the dataset gets too large, I seem to run out of memory.
W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (mklcpu) ran out of memory trying to allocate 4.47GiB (rounded to 4804430848)
Current allocation summary follows.

I've omitted the summary for brevity, but can add it in if necessary. I'm unsure of how to get around this. I've tried using tf.split to divide up the predicted and true values into smaller tensors, then passing them to the loss function separately and concatenating, but testing with a smaller dataset shows that the results don't match when I compare them to the full versions. I'm honestly stumped on how best to work around this and would be open to any recommendations.


texts: I have a txt file where values are randomly separated, sometimes there is a single space, and sometimes there are many. I am importing the file into pandas with pd.read_csv, but I get only one column in my DataFrame.  How can I define the sep parameter to read the values properly?


texts:
I have just run a Linear regression model on the Dataset having 7 independent variable and 1 target variable. Below is the R squared and MSE values.

Mean squared error for training set : 36530921.0123
$R^2$ value for training set : 0.7477

Can anybody please give me some tips to increase the efficiency of this model.

Edit: I have just implemented the same problem using Linear regression with Normalization of the features.
I got the below output:
Mean squared error for training set : 5.468490570335696e-10
R2 value for training set : 0.9275088299658416
Mean squared error for training set : 4.111793316375822e-10
R2 value for training set : 0.9342888671422529

So can we consider normalizing the dataset to get better accuracy ?


texts: What is the Most Efficient Tool in Python for row-wise manipulation of data?

texts:
I'm doing a lot of work that requires operations to be performed across rows, using the data in that rows's columns on other columns in the row.
I recently had to do some processing on a 1.2 million row dataset (27 columns, 300 Mb) that required row-wise modification of ~ 300,000 rows. Some transactions were being split across multiple records and I wanted to aggregate them. This involved collapsing them upwards with lazy deletion, summing 5 columns and doing a weighted average on a 6th.
I at first attempted to use Pandas.
Pandas spent 12+ hours to go through ~ 80,000 rows (the machine I'm using at the office is trash).
I could maybe see that if I were dropping the rows as I went and it had to resettle 300 Mb of arrays in memory after each row, but it never even got around to deleting rows since that was set to happen at the end.
I've generally seen that the sentiment around Pandas seems to be "Don't iterate over a dataframe's rows."
I'm not asking for how to make it work in Pandas, the purpose of the above was to provide context as to why I believe that Pandas isn't a viable option for what I was trying to do.
I ended up going for a basic list implementation and it ran in 5-10 seconds.
In this case, the list implementation worked. Since I had the indices of the rows in the dataframe, I avoided the O(N) look-up time pitfall.
So, while Pandas is highly optimized for column-wise operations, it's evidently not really intended to perform granular row modification, en masse (Case in point, look at the big bolded warning in the iterrows documentation).
So here's the question:
Assuming you have a non-uniform row of data, with different sized columns and data types, what is the most efficient data structure/ library to perform a large number of row-wise modifications?
What about if all the columns were floats?


texts: Gaussian distribution in python without using libraries

texts:
I am implementing Gaussian distribution of a variable, but it gives multiple bell shapes. It should be a single bell shape. Below is my code and plot.

def gaussian_transform(arr):

    u=arr.mean()

    sig=arr.std()

    fd=[]

    a=0
    pi=math.pi
    for i in arr:

        a=((i-u)**2)/(2*(sig**2))
        e=math.exp(-a)
        fd.append((1/(sig*math.sqrt(2*pi))) * e)

    return fd

y=gaussian_transform(df.Age)

plt.plot(df.Age,y)

plt.ylabel('some numbers')
plt.show()

`


texts: what is the complexity of a bidirectional recurrent neural network?

texts:
In particular, what is the complexity of a bi-directional recurrent neural network taking into account the variants of LSTM and GRU as well for training?
I am hoping if I can get links to some additional research papers which talk or have mentioned the computational complexity of these methods in their works. I have been searching, but haven't come across anything meaningful till now


texts:
I tried to predict on training set but i got accuracy of 100%. However on the testset, i got an accuracy of 62%. Should i be worried of getting high accuacies on training dataset using caret?
why accuracy on training dataset is 100% using random forest caret , does it indicate overfitting?


texts: Why the invariant reward helps training?

texts:
I am new to Machine Learning, and I am trying to solve MountainCar-v0 using Q-learning. I can solve the problem now, but I am still confused.

According to the MountainCar-v0's Wiki, the reward remains -1 for every step, even if the car has reached the destination. How does the invariant reward help the agent learn? If every step gives the same reward, how can the agent tell if it is a good move or a bad move?

Thanks in advance!


texts:
Apart from Luhn algorithm, can we use a machine algorithm to check if last digit of credit card number is valid? I see this a multi-class classification problem but unable to find the right algorithm to solve it.

Am basically looking out for an approach to find check digit for any sequence, where check digit generation algorithm is not known - Credit card is just an example.

There are multiple cases where the number issuing authority does not disclose the check digit algorithm. For e.g. GSTIN in India - cleartax.in/s/know-your-gstin OR PAN in India - en.wikipedia.org/wiki/Permanent_account_number If we can try different ML algorithms for valid set of such sequences, we might be able to create a verification model for such cases


texts: One part of my loss function overfits. How do I fix this?

texts:
I am working on an object detection problem where the final loss that is being optimized is the sum of an L2 loss (for the error in the predicted w, h values), and three binary cross entropy losses (for the predicted class error, the object/no-object confidence error and the x,y coordinate error).

I see that during the training phase, the x,y and w,h losses are overfitting (validation losses go up while the training losses go down). The other losses all do seem to be optimized normally.

How do I solve this, and why am I seeing this? The overall loss seems to be going down, as the trend is influenced by the confidence loss which is substantially higher than the rest of the losses.

For a bit more context, I'm using a YOLOv3-based object detection setup.

Edit: Attaching a picture showing the xy training and validation losses overfitting. The wh losses behave in a similar fashion.

Edit 2: The setup is to detect objects on a custom dataset of ~9000 train images and ~1000 validation images. I believe the dataset and its distribution is proper because it was validated using other YOLO-based object detection models with a couple of different backend networks, all of which gave me high accuracies. This particular problem began when I switched to a different backend network with a similar number of parameters and layers as one of the previous ones that were tested and deemed proper.


texts: It seems that the output of sklearn.metrics.pairwise.euclidean_distances is different to the formula on doc

texts:
The doc of sklearn.metrics.pairwise.euclidean_distances() gives this formula

  dist(x, y) = sqrt(dot(x, x) - 2 * dot(x, y) + dot(y, y)).

Apply this formula to this example

X = [[0, 1],
     [2, 3]]

Y = [[1, 2],
     [3, 4]]

np.dot(X,X) - 2*np.dot(X,Y) + np.dot(Y,Y)

gives this result

array([[ 3,  5],
       [-1,  1]])

whilst calling sklearn.metrics.pairwise.euclidean_distances()

euclidean_distances(X , Y, squared = True)

gives

array([[ 2., 18.],
       [ 2.,  2.]])

It seems that the output of euclidean_distances() is not consistent to the formula from the doc.


texts:
For combining the results of different measurements, I want to calculate their arithmetic mean. Imagine the following dataset:

id;attribute;time
A;AB;1
B;AB;2
C;AB;3
B;AB;4
B;AB;5
A;AB;6

I want to combine all results with id A and get their mean. The expected dataset looks like the followind. Be aware, that the value of the in between attribute, is not important, I've chosen the first entry.

id attribute     time
A  AB            3.500000
B  AB            3.666667
C  AB            3.000000

The code below does the right thing, but I'm wondering if there is a better solution for this kind of problem.

# Read in the data
data <- data.frame(read_delim("~/...", ";", escape_double = FALSE, trim_ws = TRUE))
# Visualize head of the data
head(data)

# Define function to calculate the mean
calculate_mean = function(data){

  # Prepare empty result data set
  result = data.frame(id=character(),
                      attribute=character(),
                      time=integer())

  # Iterate over specified id <-- Change this one to the one you want to iterate over
  iterator = unique(data$id)

  # Fill result dataset with mean
  for(i in iterator){
    subset = subset(data, data$id == i)
    arithmetic_mean = mean(subset$time)
    result = rbind(result, data.frame(subset[1,]$id,
                                      subset[1,]$attribute,
                                      arithmetic_mean))
  }

  # Set colnames properly
  colnames(result) = colnames(data)

  return(result)
}

# Call function
result = calculate_mean(data)

# Display the result
result

Do you have an idea? Thanks for your support.


texts:
How would I go about attaching a probability to the prediction outputted by a Gaussian Naive Bayes model ?
I'm asking because the predict_proba function U can use with sklearn's Gaussian Naive Bayes classifier only gives me zeros or one, which is not what I want.


texts: What is the difference between semantic segmentation, object detection and instance segmentation?

texts: I'm fairly new at computer vision and I've read an explanation at a medium post, however it still isn't clear for me how they truly differ.


texts: plot show relationship between independent variable and dependent variable(Binary)

texts:
My dependent variable is binary. Most of my independent variables are not. I am at the exploratory stage right now.

Y     X1     X2
0     23    0
1     29    1
0     15    1
1     40    0
1     25    1
0     22    1
This is just a portion of my data.

I was thinking scatter plot to find out the relationship between Y X1 and X2.
What other plots I can do to see the relationship more clearly.


texts: Why is my validation loss going up while my validation accuracy also goes up?

texts:
Scenario:
I've been training a CNN for the cifar10 dataset.
I'm using tensorflow, and a CNN with 12 conv layers and 1 dense layer before a softmax dense layer.
I'm using data augmentation as well with batch normalization.

After a few hundred epochs I archieved a maximum of 92.73 percent accuracy on the validation set.

My problem:

Validation loss goes up slightly as I train more.
While validation loss goes up, validation accuracy also goes up.

Example:

One epoch gave me a loss of 0.295, with a validation accuracy of
90.5%. My best epoch for validation accuracy gave me 92.73% with a validation loss of 0.33.

Question:

Why is my validation accuracy increasing while my validation loss is
going up?
Should I use a loss metric diferent to cross_entropy?


texts:
I have measured stress at 1 million points inside a material at 100 time steps. I have made a probability distribution plot for three time steps and I see that the evolution of stress looks like this:

I was wandering if I can use some machine learning algorithm to detect this behavior early on. I have tried PCA, even kPCA to reduce the dimensions but they just give me a random scatter. Also tried k-means and tSNE to segregate different clusters based on their magnitude, but they give a cloud of points.

Also, someone told me that because the magnitude of stress in each point is shifting over time (i.e. some points are stagnating and others are stressing more as time passes), the stress magnitude is not a good candidate for a "feature". Is this true? Should a "feature" stay constant over time?


texts: What do you call a feature that always has the same value?

texts:
Is there a standard term for a feature that always has the same value, i.e. that can be discarded without loss of information?

For example I am trying to classify cats vs dogs, and every example in my training set has has_two_eyes=true.

I am thinking something like "useless," "redundant," "constant," or "degenerate," but I don't know what the standard terminology is here.


texts: Difference between normalization and zero centering

texts: I am working on some pre-processing for lung CT images. I see a nice tutorial in here. Two of them are normalization and zero centering. I wonder what is the difference between these two steps? If one normalizes the image pixel values to range [0, 1], is it have a benefit to also do zero centering? Is there a difference if I do normalization first or zero centering first?


texts: how to input the data set in to a word2vec by keras?

texts:
I am new in using word2vec model, as a result, I do not know how I can prepare my dataset as an input for word2vec?
I have searched a lot but the datasets in tutorials were in CSV format or just one txt file, but my dataset is in this structure: 2 folders one of these is blood cancer and the other one is breast cancer.
each folder contains 1000 txt files which contain 40 sentences.
I do not have any idea about I can create a vocabulary as an input for the word2vec model in keras with tensorflow backend?
I use python 3.5 in ubuntu 17.10
Any guidance will be appreciated.


texts: Masked Language Modeling on Domain-specific Data

texts:
My goal is to have a language model that understands the relationships between words and can fill the masks in a sentence related to a specific domain. At first, I thought about pretraining or even training a language model(like BERT) from scratch, but unfortunately, my data isn't that big to help the previous model learn new connections, let alone learn the embeddings from scratch.
Now what I have in mind is creating a transformer model with my own vocabulary which consists of words in my domain-specific data (after separating them with spaces and not using transformer tokenizers). This way the vocab size would be smaller and the positions and relations would be learned faster and more easily. Although I'm a bit confused about implementation.
Can I use this architecture (that is for NMT) and give plain text for both the input and output? or should I mask some tokens in the input and give the complete sentence as the label?
Any other suggestions?


texts:
So, i was trying to improve mr LR model performing multiple linear regression on a dataset. I had a categorical variable region

Region(variable):
Midwest
Northeast
South
West

I made dummy variable for each of them and it did improved my model a bit.

Previous Model Summary

After adding these variable(Which i made using a different variable)

I am getting NA's in the coefficient for west, which i don't understand why. can someone explain?


texts:
Is there a correct order I should put data transformations into a pipeline using Sklearn?

Currently I have these items in my pipeline;

Feature selection, skew removal, scaling, outlier removal, oversampling and the estimator.

Is there a right or wrong way to do it? Or is there no free lunch?


texts: multilabel program: separate classifiers vs combined classifiers

texts:
I am building a program to implement a solution for multilabel classification. It's an interesting problem because most of the examples (~90%) actually only have one class, but some have multiple classes.

Additionally there are certain classes which we know to be mutually exclusive, i.e. if an example is in class A then it cannot be in class B, and vice-versa.

The way it's currently implemented is to identify all the classes which are mutually exclusive and place them within their own classifier, then build a separate classifier for each of the others:

Classifier 1: Predicts A (or OTHER)
Classifier 2: Predicts B (or OTHER)
Classifier 3: Predicts C,D,E, or F (or OTHER)

When the classifier runs, it gets predictions from all 3 classifiers and then combines them. So if classifier 1 predicts A, and classifier 2 and 3 both predict OTHER, then the result would be A. If instead classifier 2 had predicted B, then the result would be A,B.

So my questions are:

Does this seem like a reasonable implementation to you?
It seems like the downside is that we could end up with A and B too many times when really we wanted them to be exclusive more often. In that case we could try to improve things with a classifier that controls the degree of exclusivity by predicting A, B, A-and-B instead. But is that likely to represent an improvement, or would it be unnecessary?

In an ideal world if the A and B classifiers are functioning correctly I think it would be unnecessary, but I don't know if combining the decision into a single classifier would allow the classifier to learn the boundaries "better" in some way.


texts:
I'm trying to train a model to predict the final cost of a product being developed over a few months. I have historical data of similar products which will be used for training the model.
Some of the features include %planned, %developed, %purchase orders raised, pending purchase orders, material cost thus far, time period (sampled daily), overhead costs, total cost.
The aim is to predict the final cost of the project given the current status. I'm familiar with LSTMs and know that this is a multivariate-multistep prediction.
However the question remains,

the project end date is not fixed, so how do we select a window in this scenario. Some projects may last a month, others last several months.

now since the end date is not fixed we don't know mow far in the future we have to predict. So number of steps to be predicted changes as well. I figured I'd build a regression model to predict the project completion date according to the current project progress and then use that to predict the future time steps.

Also every LSTM model I've worked on and seen so far have a fixed window for prediction (predict the next 7 time steps). In this situation assuming that I use point #2 to define how many steps to predict in the future the number changes. Initially predict first 30 time steps, as we progress, predict next 12 time steps, etc. , how do I go about building a model for the same?

One approach I could think of is train the model to predict just one time step and loop over the current prediction for as many time steps as required. Now I do understand that this will decrease the accuracy tremendously if we predict too many time steps based on current predictions, (compounding error).
Let me know if any clarification is required in the comments. Thanks in advance.


texts: Validation of a model generated by automated deep learning system

texts:
How would one go about selecting the validation set used to evaluate trained models by an automated system, in order to ensure that each new model is at least as good as, or better than the previous model?

Lets say that in this case we have a regression problem, and we can depend on a single performance metric such as RMSE.

I'm working on a program that automatically trains, tests and if the tests passed, deploys a new deep learning model every x new data samples/time.
Each training is done on the original dataset (labeled data) and the new dataset which is labeled by predictions of the previous model.

These were the options I came up with and some pros/cons.

Option A:
Create a validation set once, always apply this set
+ Always have a consistent benchmark
- Validation set might get 'out of date' in cases where the relevancy of data might change with time

Option B:
Create an initial validation set, add X percentage of new training samples to the original set
+ Consistent time spread of samples

Option C:
Create a random validation set every training
+ Benchmark is most likely to contain a relevant set of samples (That are in data)
- Test results can vary each training session

Option D:
Cross-validation
+ Benchmark is most likely to contain a relevant set of samples (That are in data)
+ Likely least variance between each training session
- Computationally expensive, even more so for deep learning models


texts: Cluster analysis as an associative model?

texts:
I have a set of data with many samples and many features, but where half of the data is missing one variable (call it A), which is composed of four categories. Based on the half of data which has A, I want to know what category the samples without A would most likely be in if they did have A.

I could build a classifier based on the data with A, and predict the data without A (this is the best route IMHO).

But I'm wondering, out of curiosity, if this method could also be a very, very, very rough way of doing something similar:

Cluster the data which has A into the same number of clusters as categories in A (in this case four).
Check for an an association between the clusters and the categories in A (using a frequency table and chi-square test).
If there is an association, run the data without A through the clustering model to figure out what category of A it's most likely associated with (based on what cluster it is in).

Thoughts?


texts: I know some techniques for augmenting data when images are used, but I don't know if there are any such techniques specifically catered to videos. Since videos have an additional temporal dimension compared to images I was wondering if there were any techniques that augments the video data using the view points and the motion information etc


texts: How does generalised advantage estimation work?

texts:
I've been trying to add GAE to my A2C implementation for a while now, but I' can't quite seem to grok how it works.

My understanding of it, is that it reduces the variance of the advantage estimation function by kind of 'averaging out' (or generalising) the advantages based off the values in the rollout.

I tried to run through the maths on my own, and in the end I just had one advantage for the whole rollout, is this right? Normally, we'd have one advantage for each timestep in the rollout.

Can anyone provide an explanation on the intuition of GAE?


texts:
I am having a hard time understanding the strategy for inputting the color. Most tutorials on RBMs only train grayscale images.

If the image is grayscale, the input units can be binary, and I can normalize the gray scale value to [0,1], and then treat them like probabilities in the input layer. Or whiten the dataset and use Gaussian units in the input layer.

How do I treat color images? Obviously, the input units cannot be binary - unless I replicate the units for each of the three color channels? Or what is the popular strategy?


texts: What does it mean to 'gradient check a few dimensions for every separate parameter'?

texts:
In lecture notes for cs231n while discussing checking analytical gradient with numerical gradient the paragraph says this:

Check only few dimensions. In practice the gradients can have sizes of million parameters. In these cases it is only practical to check some of the dimensions of the gradient and assume that the others are correct.
Be careful: One issue to be careful with is to make sure to gradient check a few dimensions for every separate parameter. In some applications, people combine the parameters into a single large parameter vector for convenience. In these cases, for example, the biases could only take up a tiny number of parameters from the whole vector, so it is important to not sample at random but to take this into account and check that all parameters receive the correct gradients.

I understand to check only a few dimensions but what does the part after be careful mean. I fail to understand that warning. can somebody explain it?


texts: What are the best sites to display your data science skills & projects? Obviously there's kaggle, but any others that recruiters look at?


texts: How to interpret the AUC score in this case?

texts:
I just run a random forest model on a imbalance dataset. I got the set of AUC and the confusion matrix. The AUC seemed not bad but actually the model predict every instance as positive. So how it happened and how to use AUC properly?

The ROC Curve as below:

I plot out the predicted probability of positive class in test set. The probability was within a tight range (0-0.4).


texts: Pandas - read CSV with spanish characters

texts:
I have DataFrame which has column with spanish regions, when I import the .csv as following

df = pd.read_csv('...',delimiter=';',decimal=',')

And print the content of column, I see this: Castilla y Le�n.

I could not find a way how to fix it, but I am assuming there is smart way how to do it.


texts: How to convert input numpy data to tensorflow tf.data to train model in tensorfow?

texts:
I am working on an image classification problem using TensorFlow. I have converted my input image dataset and label into NumPy data but it takes more time and more ram to load all the data into memory because I have 90K images. I would like to use TensorFlow data API using tf.keras.preprocessing.image_dataset_from_director
Here's my current code to train NumPy data. I want to convert this code into tf.data(tf.keras.preprocessing.image_dataset_from_director) to train my huge dataset.
INIT_LR = 1e-4
EPOCHS = 20
BS = 32

# grab the list of images in our dataset directory, then initialize
# the list of data (i.e., images) and class images
print("[INFO] loading images...")
imagePaths = list(paths.list_images(args["dataset"]))
data = []
labels = []

# loop over the image paths
for imagePath in imagePaths:
    # extract the class label from the filename
    label = imagePath.split(os.path.sep)[-2]

    # load the input image (224x224) and preprocess it
    image = load_img(imagePath, target_size=(224, 224))
    image = img_to_array(image)
    image = preprocess_input(image)

    # update the data and labels lists, respectively
    data.append(image)
    labels.append(label)

# convert the data and labels to NumPy arrays
data = np.array(data, dtype="float32")
labels = np.array(labels)

# perform one-hot encoding on the labels
lb = LabelBinarizer()
labels = lb.fit_transform(labels)
labels = to_categorical(labels)

# partition the data into training and testing splits using 75% of
# the data for training and the remaining 25% for testing
(trainX, testX, trainY, testY) = train_test_split(data, labels,
    test_size=0.20, stratify=labels, random_state=42)

# construct the training image generator for data augmentation
aug = ImageDataGenerator(
    rotation_range=20,
    zoom_range=0.15,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.15,
    horizontal_flip=True,
    fill_mode="nearest")

# load the MobileNetV2 network, ensuring the head FC layer sets are
# left off
baseModel = MobileNetV2(weights="imagenet", include_top=False,
    input_tensor=Input(shape=(224, 224, 3)))

# construct the head of the model that will be placed on top of the
# the base model
headModel = baseModel.output
headModel = AveragePooling2D(pool_size=(7, 7))(headModel)
headModel = Flatten(name="flatten")(headModel)
headModel = Dense(128, activation="relu")(headModel)
headModel = Dropout(0.5)(headModel)
headModel = Dense(2, activation="softmax")(headModel)

# place the head FC model on top of the base model (this will become
# the actual model we will train)
model = Model(inputs=baseModel.input, outputs=headModel)

# loop over all layers in the base model and freeze them so they will
# *not* be updated during the first training process
for layer in baseModel.layers:
    layer.trainable = False

# compile our model
print("[INFO] compiling model...")
opt = Adam(lr=INIT_LR, decay=INIT_LR / EPOCHS)
model.compile(loss="binary_crossentropy", optimizer=opt,
    metrics=["accuracy"])

# train the head of the network
print("[INFO] training head...")
H = model.fit(
    aug.flow(trainX, trainY, batch_size=BS),
    steps_per_epoch=len(trainX) // BS,
    validation_data=(testX, testY),
    validation_steps=len(testX) // BS,
    epochs=EPOCHS)

# make predictions on the testing set
print("[INFO] evaluating network...")
predIdxs = model.predict(testX, batch_size=BS)

# for each image in the testing set we need to find the index of the
# label with corresponding largest predicted probability
predIdxs = np.argmax(predIdxs, axis=1)

# show a nicely formatted classification report
print(classification_report(testY.argmax(axis=1), predIdxs,
    target_names=lb.classes_))

# serialize the model to disk
print("[INFO] saving mask detector model...")
model.save(args["model"], save_format="h5")


texts:
I have a project in machine learning in which I need to analyze a
  curriculum vitae. for that I have to write a python program. It uses
  basic techniques of Natural Language Processing like word
  parsing,chunking,reg ex parser. If you run the algorithm you can
  easily capture information like name,email id,address,educational
  qualification,experience in seconds from a large number of documents.
  but I have a confusion between two methods LSTM (RNN) and NER? What do
  you think is the best method?